{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model analysis and errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is for collecting examples of model reasoning, but also errors. It is used by the group of ThoughSource to save and exchange findings.\n",
    "Please document your examples exactly such that we can reproduce them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9187b1f2-f4dd-49c4-b8e7-1debf302223d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cot.dataloader import Collection\n",
    "from cot.generate import generate_and_extract, FRAGMENTS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please copy the template below and then make an analysis or error report based on its structure.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Template for reports and reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading worldtree...\n"
     ]
    }
   ],
   "source": [
    "# provide which dataset you loaded\n",
    "collection = Collection([\"worldtree\"], verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# provide config file, also just copying it here without the comments is enough.\n",
    "config={\n",
    "    \"idx_range\":(0,1), # Determines which indices the generate_and_extract routine is applied to, Default: None (All items are used)\n",
    "    \"debug\": True, # Determines whether an api is called or a mock is returned, used for debugging, Default: True (api is not used)\n",
    "    \"instruction_keys\": [\"qa-01\"], # Determines which instructions are used from fragments.json, Default: None (All used)\n",
    "    \"cot_trigger_keys\": [\"kojima-01\"], # Determines which cot triggers are used from fragments.json, Default: None (All are used)\n",
    "    \"answer_extraction_keys\": [\"kojima-01\"], # Determines which answer extraction prompts are used from fragments.json, Default: None (All are used)\n",
    "    \"author\" : \"Konstantin\", # Name of the person responsible for generation, Default: \"\"\n",
    "    \"api_service\": \"huggingface_hub\", # Name of the API called (\"openai\", \"huggingface_hub\"), Default: \"openai\"\n",
    "    \"engine\": \"google/flan-t5-xl\", # Name of the engine used (for \"huggingface_hub\" use for example \"google/flan-t5-xl\"), Default: \"text-davinci-002\"\n",
    "    \"temperature\": 0, # Name of the person responsible for generation, Default: 0\n",
    "    \"max_tokens\": 512, # Maximum lenght of output generated by the model, Default: 128\n",
    "    \"api_time_interval\": 1.0, # Pause between two api calls in seconds, Default: 1.0\n",
    "    \"warn\": False,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "277bc208",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_samples: 3, n_instruction_keys: 1, n_cot_trigger_keys: 1, n_answer_extraction_keys: 1\n"
     ]
    }
   ],
   "source": [
    "# generate examples\n",
    "collection.generate(\"worldtree\", config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55d43612",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'annotation': [],\n",
       "   'answers': [{'answer': '\\nB) Earth rotates on its axis.',\n",
       "     'answer_extraction': 'kojima-01',\n",
       "     'correct_answer': None}],\n",
       "   'api_service': 'openai',\n",
       "   'author': 'Konstantin',\n",
       "   'comment': '',\n",
       "   'cot': '\\nThe Sun appears to move across the sky every day.\\n\\nThe most likely explanation is that Earth rotates on its axis.',\n",
       "   'cot_trigger': 'kojima-01',\n",
       "   'date': '2022/11/17 10:45:43',\n",
       "   'instruction': 'qa-01',\n",
       "   'model': {'max_tokens': 512, 'name': 'text-davinci-002', 'temperature': 0},\n",
       "   'templates_version': '0.01'}],\n",
       " []]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# save your relevent outputs by copying them to the markdown field below\n",
    "# collection[\"worldtree\"][\"train\"]['generated_cot'] \n",
    "collection[\"worldtree\"][\"train\"][list(config['idx_range'])]['generated_cot']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d43612",
   "metadata": {},
   "source": [
    "**Copy the relevant outputs here:**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d43612",
   "metadata": {},
   "source": [
    "{'annotation': [],\n",
    "   'answers': [{'answer': '\\nB) Earth rotates on its axis.',\n",
    "     'answer_extraction': 'kojima-01',\n",
    "     'correct_answer': None},\n",
    "    {'answer': '\\nB) Earth rotates on its axis.',\n",
    "     'answer_extraction': 'kojima-02',\n",
    "     'correct_answer': None}],\n",
    "   'api_service': 'openai',\n",
    "   'author': '',\n",
    "   'comment': '',\n",
    "   'cot': '\\nThe Sun appears to move across the sky every day.\\n\\nThe most likely explanation is that Earth rotates on its axis.',\n",
    "   'cot_trigger': 'kojima-01',\n",
    "   'date': '2022/11/15 13:19:25',\n",
    "   'instruction': 'qa-01',\n",
    "   'model': {'max_tokens': 512, 'name': 'text-davinci-002', 'temperature': 0},\n",
    "   'templates_version': '0.01'}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d43612",
   "metadata": {},
   "source": [
    "**Description of issue:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the field for your interpretation or error report."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model uses B) C) D) in CoT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading worldtree...\n"
     ]
    }
   ],
   "source": [
    "# provide which dataset you loaded\n",
    "collection = Collection([\"worldtree\"], verbose=False)\n",
    "collection = collection.select(split=\"train\", random_samples=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# provide config file, also just copying it here without the comments is enough.\n",
    "config={\n",
    "    \"idx_range\":(1722,1723),\n",
    "    \"debug\": False,\n",
    "    \"instruction_keys\": ['qa-01'],\n",
    "    \"cot_trigger_keys\": ['kojima-01'],\n",
    "    \"answer_extraction_keys\": ['kojima-A-D'],\n",
    "    \"api_service\": \"huggingface_hub\", # Name of the API called (\"openai\", \"huggingface_hub\"), Default: \"openai\"\n",
    "    \"engine\": \"google/flan-t5-xl\", # Name of the engine used (for \"huggingface_hub\" use for example \"google/flan-t5-xl\"), Default: \"text-davinci-002\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "277bc208",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "            You are about to \u001b[1m call an external API \u001b[0m in total 2 times, which \u001b[1m may produce costs \u001b[0m.\n",
      "            Number API calls for CoT generation: n_samples 1 * n_instruction_keys 1 * n_cot_trigger_keys 1\n",
      "            Number API calls for answer extraction: n_samples 1 * n_instruction_keys 1 * n_cot_trigger_keys 1 * n_answer_extraction_keys 1\n",
      "            Do you want to continue? y/n\n",
      "            \n"
     ]
    }
   ],
   "source": [
    "# generate examples\n",
    "collection.generate(\"worldtree\", config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "55d43612",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'templates_version': '0.01',\n",
       "   'instruction': 'qa-01',\n",
       "   'cot_trigger': 'kojima-01',\n",
       "   'prompt_text': \"Answer the following question through step-by-step reasoning.\\nSharpening a pencil and tearing paper are examples of physical changes. Which statement describes why these are physical changes?\\nA) There is a change in how the objects are used.\\nB) There is a change in the appearance of the objects.\\nC) There is a change in the materials from which the objects are made.\\nD) There is a change in both the appearance of the objects and the materials from which they are made.\\nAnswer: Let's think step by step.\\n\",\n",
       "   'answers': [{'answer_extraction': 'kojima-A-D',\n",
       "     'answer_extraction_text': \"Answer the following question through step-by-step reasoning.\\nSharpening a pencil and tearing paper are examples of physical changes. Which statement describes why these are physical changes?\\nA) There is a change in how the objects are used.\\nB) There is a change in the appearance of the objects.\\nC) There is a change in the materials from which the objects are made.\\nD) There is a change in both the appearance of the objects and the materials from which they are made.\\nAnswer: Let's think step by step.\\nB) There is a change in the appearance of the objects. C) There is a change in the materials from which the objects are made. D) There is a change in both the appearance of the objects and the materials from which they are made. So, the final answer is B.\\nTherefore, among A through D, the answer is\\n\",\n",
       "     'answer': 'B.',\n",
       "     'correct_answer': None}],\n",
       "   'cot': 'B) There is a change in the appearance of the objects. C) There is a change in the materials from which the objects are made. D) There is a change in both the appearance of the objects and the materials from which they are made. So, the final answer is B.',\n",
       "   'author': '',\n",
       "   'date': '2022/11/24 16:46:05',\n",
       "   'api_service': 'huggingface_hub',\n",
       "   'model': \"{'name': 'google/flan-t5-xl', 'temperature': 0, 'max_tokens': 128}\",\n",
       "   'comment': '',\n",
       "   'annotation': []}],\n",
       " []]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save your relevent outputs by copying them to the markdown field below\n",
    "# collection[\"worldtree\"][\"train\"]['generated_cot'] \n",
    "collection[\"worldtree\"][\"train\"][list(config['idx_range'])]['generated_cot']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d43612",
   "metadata": {},
   "source": [
    "**Copy the outputs here:**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d43612",
   "metadata": {},
   "source": [
    "[[{'templates_version': '0.01',\n",
    "   'instruction': 'qa-01',\n",
    "   'cot_trigger': 'kojima-01',\n",
    "   'prompt_text': \"Answer the following question through step-by-step reasoning.\\nSharpening a pencil and tearing paper are examples of physical changes. Which statement describes why these are physical changes?\\nA) There is a change in how the objects are used.\\nB) There is a change in the appearance of the objects.\\nC) There is a change in the materials from which the objects are made.\\nD) There is a change in both the appearance of the objects and the materials from which they are made.\\nAnswer: Let's think step by step.\\n\",\n",
    "   'answers': [{'answer_extraction': 'kojima-A-D',\n",
    "     'answer_extraction_text': \"Answer the following question through step-by-step reasoning.\\nSharpening a pencil and tearing paper are examples of physical changes. Which statement describes why these are physical changes?\\nA) There is a change in how the objects are used.\\nB) There is a change in the appearance of the objects.\\nC) There is a change in the materials from which the objects are made.\\nD) There is a change in both the appearance of the objects and the materials from which they are made.\\nAnswer: Let's think step by step.\\nB) There is a change in the appearance of the objects. C) There is a change in the materials from which the objects are made. D) There is a change in both the appearance of the objects and the materials from which they are made. So, the final answer is B.\\nTherefore, among A through D, the answer is\\n\",\n",
    "     'answer': 'B.',\n",
    "     'correct_answer': None}],\n",
    "   'cot': 'B) There is a change in the appearance of the objects. C) There is a change in the materials from which the objects are made. D) There is a change in both the appearance of the objects and the materials from which they are made. So, the final answer is B.',\n",
    "   'author': '',\n",
    "   'date': '2022/11/24 16:46:05',\n",
    "   'api_service': 'huggingface_hub',\n",
    "   'model': \"{'name': 'google/flan-t5-xl', 'temperature': 0, 'max_tokens': 128}\",\n",
    "   'comment': '',\n",
    "   'annotation': []}],\n",
    " []]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d43612",
   "metadata": {},
   "source": [
    "**Description of issue:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model uses the letters B) C) D) to structure its text in the CoT, which makes no sense."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.10 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "a19cb823e24c98ee05a9cfa4a3a579b5d56d4c1a735f2a12456750b95a1e155e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
